Thresholded Lasso for High Dimensional Variable Selection

Given n noisy samples with p dimensions, where n " p, we show that the multi-step thresholding procedure based on the Lasso – we call it the Thresholded Lasso, can accurately estimate a sparse vector β ∈ R in a linear model Y = Xβ + ", where Xn×p is a design matrix normalized to have column #2-norm √ n, and " ∼ N(0,σIn). We show that under the restricted eigenvalue (RE) condition (BickelRitov-Tsybakov 09), it is possible to achieve the #2 loss within a logarithmic factor of the ideal mean square error one would achieve with an oracle while selecting a sufficiently sparse model – hence achieving sparse oracle inequalities; the oracle would supply perfect information about which coordinates are non-zero and which are above the noise level. We also show for the Gauss-Dantzig selector (Candès-Tao 07), ifX obeys a uniform uncertainty principle, one will achieve the sparse oracle inequalities as above, while allowing at most s0 irrelevant variables in the model in the worst case, where s0 ≤ s is the smallest integer such that for λ = √ 2 log p/n, ∑p i=1 min(β 2 i ,λ σ) ≤ s0λσ. Our simulation results on the Thresholded Lasso match our theoretical analysis excellently. keyword. Linear regression, Thresholded Lasso, Lasso, #1 regularization, #0 penalty, multiple-step procedure, Gauss-Dantzig Selector, ideal model selection, oracle inequalities, restricted orthonormality, Restricted Eigenvalue condition, statistical estimation, thresholding, linear sparsity, random matrices